{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "LfCmmM4_bnov"
      },
      "source": [
        "# Postprocess results from Earth Engine\n",
        "\n",
        "Author: Frederik Kratzert\n",
        "Contact: kratzert@google.com\n",
        "\n",
        "This notebook is part of the [Caravan publication](https://www.nature.com/articles/s41597-023-01975-w) and is the second of two notebooks that can be used to extend the dataset to new basins. This notebook is expected to be run locally (or on your own server infrastructure) and requires the outputs of the first notebook (the downloaded data from Earth Engine). \n",
        "\n",
        "## What is Caravan?\n",
        "\n",
        "Caravan is ...\n",
        "- an open source, large-sample rainfall-runoff dataset.\n",
        "- derived entirely in the cloud (Earth Engine).\n",
        "- a dataset that only uses globally available data products (namely [ECMWF's ERA5-Land](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-land?tab=overview) and [HydroATLAS](https://www.hydrosheds.org/page/hydroatlas) by Linke et al.).\n",
        "\n",
        "Why?\n",
        "- to democratize large-sample hydrology: Anyone can extend the dataset to new regions without downloading and preprocessing large amounts of raw data. Instead, we use freely available cloud resources to do all heavy lifting in the cloud and only download the processed results.\n",
        "- to advance research on global rainfall-runoff models: So far, we have seen  different large-sample datasets (e.g. CAMELS variants) that are specific to individual countries. However, due to the use of data products with only local coverage (e.g. meteorological forcing data or static attribute maps), intercomparisons betweens these products are not straight forward.\n",
        "\n",
        "## Requirements\n",
        "\n",
        "- Python environment with Python >= 3.6 and the following libraries installed:\n",
        "  - Jupyter\n",
        "  - netCDF4\n",
        "  - Numba\n",
        "  - Pandas\n",
        "  - tqdm\n",
        "  - xarray\n",
        "  - timzonefinder\n",
        "  \n",
        "  Check the [Github repository](https://github.com/kratzert/Caravan) for an [environment.yml](https://github.com/kratzert/Caravan/environments/environment.yml) (for conda environments) or [requirements.txt](https://github.com/kratzert/Caravan/environments/requirements.txt) (pip) file.\n",
        "- Outputs of the [first notebook](https://github.com/kratzert/Caravan/code/Caravan_part1_Earth_Engine.ipynb), namely the data processed by Earth Engine(ERA5 Land time series and HydroATLAS attributes).\n",
        "- The latitude and longitude information for each gauge.\n",
        "- Daily streamflow observations for each gauge.\n",
        "\n",
        "## Making new data available to the community\n",
        "\n",
        "We envision Caravan as a dynamically growing dataset that can be extended by its users. While making a large-sample dataset available upon the intial publication, the distribution of the basins is still limited to a few regions in the world. Ideally, this will change over time. Anyone who uses Caravan and this code to extend the dataset is invited to also share the streamflow time series data with the community, even if it is just for a single (or a few) basins. By doing so, little by little, we could create a new, globally spanning open source dataset that anyone can use for their own work/research. For the moment, the [discussion forum](https://github.com/kratzert/Caravan/discussions) on the Github repository is the place to share information on data that you want to contribute. See also the [wiki](https://github.com/kratzert/Caravan/wiki) on Github for further information.\n",
        "\n",
        "## Structure of this notebook\n",
        "\n",
        "The code consists of different sections.\n",
        "\n",
        "1. General setup/imports/configurations\n",
        "2. Processing of the raw ERA5 Land time series outputs (hourly) from Earth Engine into daily aggregates in local time (of each gauge)\n",
        "3. Merging forcings with streamflow data\n",
        "4. Updating attributes to include latitude/longitude \n",
        "5. Saving processed data to disk\n",
        "\n",
        "## Citation\n",
        "\n",
        "If you use the Caravan dataset or this code, please consider referencing the original publication\n",
        "\n",
        "```bib\n",
        "@article{kratzert2023caravan,\n",
        "  title={Caravan-A global community dataset for large-sample hydrology},\n",
        "  author={Kratzert, Frederik and Nearing, Grey and Addor, Nans and Erickson, Tyler and Gauch, Martin and Gilon, Oren and Gudmundsson, Lukas and Hassidim, Avinatan and Klotz, Daniel and Nevo, Sella and others},\n",
        "  journal={Scientific Data},\n",
        "  volume={10},\n",
        "  number={1},\n",
        "  pages={61},\n",
        "  year={2023},\n",
        "  publisher={Nature Publishing Group UK London}\n",
        "}\n",
        "```\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "4WpQz9BI7W0c"
      },
      "source": [
        "# Import modules"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3vKQTujbhw2"
      },
      "outputs": [],
      "source": [
        "import shutil\n",
        "from pathlib import Path\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import xarray\n",
        "from timezonefinder import TimezoneFinder\n",
        "from tqdm.notebook import tqdm\n",
        "\n",
        "import caravan_utils"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "t5zxvfX_dyG3"
      },
      "source": [
        "# General configuration\n",
        "\n",
        "Make sure to edit the fields that are listed as mandatory for the rest of this notebook to work. All `*_FILE` and `_DIR` variables should be `Path()` objects."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "GuFI8POptH9s"
      },
      "source": [
        "- `BASIN_ID_FIELD`: Name of the attribute field in the shapefile that contains the basin id.\n",
        "\n",
        "- `BASIN_PREFIX`: A short descriptive string that is prepended to each basin id and that should be unique within the Caravan data space. E.g., we use `us` for basins from the CAMELS US dataset and `gb` for basins within Great Britain. The final name for each basin within the attribute table will be `{BASIN_PREFIX}_{GAUGE_ID}`. Note, if you already included such a prefix in the basin id field of your shapefile, leave this field as an empty string. Please also read the README in the Caravan dataset folder on details about the folder structure of the dataset.\n",
        "- `ATTRIBUTE_FILE`: `Path` object pointing to a csv file with the HydroATLAS attributes as saved by Earth Engine (first notebook).\n",
        "- `ERA5L_DIR`: `Path` object pointing to the directory that contains the csv files (`batchXX.csv`) with the ERA5-Land data as saved by Earth Engine (first notebook).\n",
        "- `NUM_POOL_WORKERS`: Number of parallel threads that are used for some processing steps. Should not be larger than the number of CPU kernels of your machine (potentially one less than that to have some free resources while the data is processed).\n",
        "- `ERA5L_BANDS`: The list of features from ERA5-Land that you want to process. Change this in case you only downloaded a subset of the bands that are the defaults in the first notebook. However, note that if you change this, you might need to adapt some parts of the code, as e.g. the climate attributes that are computed in this notebook specifically require some of the ERA5-Land bands to be present.\n",
        "- `MEAN_VARS`: List of features which are aggregated to daily resolution by computing the daily mean. Change this in case you only downloaded a subset of the bands that are the defaults in the first notebook.\n",
        "- `MIN_VARS`: Same as above, but instead of aggregating by computing the mean, the min value is used.\n",
        "- `MAX_VARS`: Same as above, but instead of aggregating by computing the mean, the max value is used.\n",
        "- `SUM_VARS`: Same as above, but instead of aggregating by computing the mean, the daily sum value is used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zQ5myxsDdSk5"
      },
      "outputs": [],
      "source": [
        "# ------------------------------ Mandatory ------------------------------------#\n",
        "BASIN_ID_FIELD = ''\n",
        "BASIN_PREFIX = ''\n",
        "ATTRIBUTE_FILE = Path(\"\")\n",
        "ERA5L_DIR = Path(\"\")\n",
        "OUTPUT_DIR = Path(\"\")\n",
        "\n",
        "# ------------------------------ Optional -------------------------------------#\n",
        "NUM_POOL_WORKERS = 8\n",
        "ERA5L_BANDS = ['temperature_2m', \n",
        "               'dewpoint_temperature_2m',\n",
        "               'volumetric_soil_water_layer_1', \n",
        "               'volumetric_soil_water_layer_2', \n",
        "               'volumetric_soil_water_layer_3', \n",
        "               'volumetric_soil_water_layer_4',\n",
        "               'surface_net_solar_radiation', \n",
        "               'surface_net_thermal_radiation', \n",
        "               'u_component_of_wind_10m', \n",
        "               'v_component_of_wind_10m', \n",
        "               'surface_pressure',\n",
        "               'total_precipitation', \n",
        "               'snow_depth_water_equivalent',\n",
        "               'potential_evaporation']\n",
        "MEAN_VARS = ['snow_depth_water_equivalent',\n",
        "             'surface_net_solar_radiation',\n",
        "             'surface_net_thermal_radiation',\n",
        "             'surface_pressure',\n",
        "             'temperature_2m',\n",
        "             'dewpoint_temperature_2m',\n",
        "             'u_component_of_wind_10m',\n",
        "             'v_component_of_wind_10m',\n",
        "             'volumetric_soil_water_layer_1',\n",
        "             'volumetric_soil_water_layer_2',\n",
        "             'volumetric_soil_water_layer_3',\n",
        "             'volumetric_soil_water_layer_4',]\n",
        "MIN_VARS = MEAN_VARS\n",
        "MAX_VARS = MEAN_VARS\n",
        "SUM_VARS = ['total_precipitation',\n",
        "            'potential_evaporation']\n",
        "\n",
        "if not all([isinstance(x, Path) for x in [ATTRIBUTE_FILE, ERA5L_DIR, OUTPUT_DIR]]):\n",
        "    raise ValueError(\"All paths to directories or files need to be Path objects.\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "usfDPYJsnvYg"
      },
      "source": [
        "# Load and unify raw Earth Engine outputs\n",
        "\n",
        "The purpose of this first part is to postprocess the raw outputs from Earth Engine into per-basin time series. Since the hourly data for multiple (hundreds) of basins can become quite large, this part of the code does the following.\n",
        "\n",
        "1. We read the Earth Engine outputs (the batch files) file by file. Each file contains data for a specific time period for all basins.\n",
        "2. We split this data by basins and store them temporary to per-basin csv files for each of the batch files.\n",
        "3. After all batch files are processed, we iterate over the basins and combine the temporary per-basin files (of each batch) into one single file with all data.\n",
        "\n",
        "This part of the code can become quite compute intensive, especially for a large number of basins, e.g. several hundreds. To make things faster, we offer a parallized implementation of the steps above. However, sometimes the multiprocessing library fails to process the data (or runs indefinitely without any progress), for no apparent reason. If this is happening to you, you can run the sequential processing code blow. \n",
        "\n",
        "All intermediate temporary files are deleted after this process and the combined hourly per-basin files are also deleted at the end of this notebook.\n",
        "\n",
        "## Parallelized implementation\n",
        "\n",
        "Make sure to adapt the `NUM_POOL_WORKERS` according to your system. If you noticed that nothing happens, e.g. the progress bar does not advance after a while or you see no temporary files being created at the `OUTPUT_DIR`, stop this code block and execute the (slower) sequential code below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "finished_preprocessing = False\n",
        "csv_files = sorted(list((ERA5L_DIR).glob('batch*.csv')))\n",
        "caravan_utils.process_earth_engine_outputs(\n",
        "    csv_files=csv_files, \n",
        "    basin_id_field=BASIN_ID_FIELD, \n",
        "    era5l_bands=ERA5L_BANDS, \n",
        "    output_dir=OUTPUT_DIR, \n",
        "    num_workers=NUM_POOL_WORKERS\n",
        ")\n",
        "print(f\"Temporary data stored at {OUTPUT_DIR / 'temp'}. Will be removed at the end of this notebook.\")\n",
        "basins = [p.name for p in (OUTPUT_DIR / \"temp\").glob('*')]\n",
        "finished_preprocessing = True"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Sequential implementation\n",
        "\n",
        "**ONLY RUN THIS CODE IF THE ABOVE CODE DID NOT FINISH SUCCESSFULLY**\n",
        "\n",
        "This code processes each csv file sequentially and therefore takes a good while longer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if not finished_preprocessing:\n",
        "    csv_files = sorted(list((ERA5L_DIR).glob('*.csv')))\n",
        "    for csv_file in tqdm(csv_files, desc='Splitting batch files by basin id.'):\n",
        "        df = caravan_utils.load_and_clean_csv_file(csv_file, BASIN_ID_FIELD, ERA5L_BANDS)\n",
        "        basins = list(set(df[\"gauge_id\"].to_list()))\n",
        "        for basin in basins:\n",
        "            if not (OUTPUT_DIR / \"temp\" / basin / f\"{csv_file.stem}.nc\").is_file():\n",
        "                caravan_utils.split_by_basin_and_save(df=df, basin=basin, output_dir=OUTPUT_DIR, filename=csv_file.stem)\n",
        "\n",
        "    basin_dirs = [p for p in (OUTPUT_DIR / \"temp\").glob('*')]\n",
        "    for basin_dir in tqdm(basin_dirs, desc='Combining batch files per basin.'):\n",
        "        caravan_utils.stack_per_basin_netcdfs(basin_dir)\n",
        "    basins = [p.name for p in basin_dirs]\n",
        "else:\n",
        "    print(\"Seems like the parallel preprocessing step finished successfully. If that is not true \"\n",
        "          \"set 'finshed_preprocessing' to False\")                "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "myf_wuTiBnyM"
      },
      "source": [
        "# Load gauge lat/lon info\n",
        "\n",
        "**USER INPUT REQUIRED**\n",
        "\n",
        "Here, you need to add code to load gauge latitude and longitude information. This is required to be able to convert the ERA5-Land time series from UTC to local time, before calculating daily statistics and finally merging it with daily streamflow data. Additionally, it would be amazing to add the name of the gauge station as well as the name of the country to this dataframe.\n",
        "\n",
        "You need to do the following:\n",
        "- Load latitude, longitude (of the gauge station), name (of the gauge station) and the country name(s) from any source.\n",
        "- Create a pandas DataFrame, called `df_metadata` that is indexed by the basin id (same id's as in the Earth Engine outputs) with the following columns:\n",
        "\n",
        "  - `gauge_lat` contains the latitude values (in WGS84).\n",
        "  - `gauge_lon` contains the longitude values (in WGS84).\n",
        "  - `gauge_name` contains the name of the gauge stations. If not available, it can be an empty string. But better would be to provide e.g. the closest village/city or the name of the river.\n",
        "  - `country` the name of the country."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TRshZuLc-UZe"
      },
      "outputs": [],
      "source": [
        "# Load your lat/lon information here. See specifications above\n",
        "df_metadata = ..."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Lsnl8SGZdjuj"
      },
      "source": [
        "## Validate loaded lat/lon data\n",
        "\n",
        "Execute this code section to test the validity of your `df_lat_lon` DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uizTvoU-dpJf",
        "outputId": "29095009-44a2-4f9f-9275-5680592728f4"
      },
      "outputs": [],
      "source": [
        "if not 'df_metadata' in locals():\n",
        "    raise RuntimeError(\"A variable 'df_metadata' does not exist\")\n",
        "\n",
        "if not isinstance(df_metadata, pd.DataFrame):\n",
        "    raise RuntimeError(\"'df_metadata' is not a pandas DataFrame\")\n",
        "\n",
        "missing_basins = [x for x in basins if x not in df_metadata.index]\n",
        "if missing_basins:\n",
        "    raise RuntimeError(\"The following basins are missing in the index of the \"\n",
        "                       f\"DataFrame: {missing_basins}\")                   \n",
        "  \n",
        "if any([x not in df_metadata.columns for x in ['gauge_lat', 'gauge_lon', 'country', 'gauge_name']]):\n",
        "    raise RuntimeError(\"Missing one (or more) of the following columns: \"\n",
        "                       \"'gauge_lat', 'gauge_lon', 'country', 'gauge_name'\")\n",
        "\n",
        "if df_metadata[['gauge_lat', 'gauge_lon', 'gauge_name', 'country']].isnull().values.any():\n",
        "    raise RuntimeError(\"There is at least one NaN value in the dataframe\")\n",
        "\n",
        "# Make sure the index is called \"gauge_id\"\n",
        "df_metadata.index.name = \"gauge_id\"\n",
        "\n",
        "print(\"All tests passed successfully.\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "u2qfQ1fv0loP"
      },
      "source": [
        "## Aggregate to daily resolution\n",
        "\n",
        "To be able to convert the hourly ERA5-Land data to daily aggregates we need to execute the following steps.\n",
        "\n",
        "1. Disaggregate ERA5-Land features that are provided as accumulates over the day (see explanation below).\n",
        "2. Convert ERA5-Land variables to units more common for hydrological applications.\n",
        "3. Shift to local time and aggregate to daily resolution (see explanation below).\n",
        "\n",
        "### Disaggregate ERA5Land features and convert units\n",
        "\n",
        "Some ERA5-Land features accumulate their values over one day (in UTC). Meaning that, e.g., the precipitation value at `01:00` contains the total precipitation that has fallen from `00:00` to `01:00` and the value of `02:00` contains the total precipitation that has fallen from `00:00` to `02:00` (and not the amount that has fallen from `01:00` to `02:00`). Because we want to convert the hourly UTC data to local time (below), we first disaggregate the cumulated features into hourly \"instantaneous\" (or hourly accumulated) values. Due to some numerical issues (on Earth Engine/ERA5-Land side), there might also be some physically implausible values, e.g., instead of 0 mm precipitation there is a very tiny negative value. We clip these cases to zero. \n",
        "\n",
        "### Unit conversion\n",
        "We convert the ERA5-Land features into units that are more common in hydrology (e.g., precipitation in mm instead of m). For details see the `era5l_unit_conversion` in `caravan_utils.py` or check the metadata that is stored in the netCDF files. \n",
        "\n",
        "### Aggregate to daily statistics\n",
        "\n",
        "Special care has to be taken when computing daily aggregates from hourly values, especially with those features that are accumulates, rather than instantenous values. Of the features included in Caravan, `total_precipitation`,`potential_evaporation`, `surface_net_solar_radiation`, and `surface_net_thermal_radiation` are accumulates, while all others are instantenous values. For the accumulates, according to the [ERA5Land documentation](https://confluence.ecmwf.int/display/CKB/ERA5-Land%3A+data+documentation#ERA5Land:datadocumentation-accumulationsAccumulations), the value that is stored at e.g. 1981-01-01 00:00 is the value that is accumulated over the previous hour _up to_ 00:00. This means that this value should not be included when computing the daily statistics for 1981-01-01. The 24 data points that should be included to compute e.g. the daily statistics for 1981-01-01 are from 1981-01-01 01:00 until (including) 1981-01-02 00:00. We need to consider this when we use the resampling function of the pandas library, which does (by defaul) resample by date, which would be wrong in this case. We can mitigate that by passing an `offset=pd.Timedelta(hours=-1)` to the resample function. For the instantenous values it doesn't really make a difference if the steps 00:00 to 23:00 are used, or also the values 01:00 to 00:00 (==24:00). Therefore, we chose to take the same timesteps as for the accumulates."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "9ec4d400994e4034975bd9a9de1a9622",
            "28bcbfba0dad4138aeafef2fe138c119",
            "f416a0e2889d4d0897c4ba8f22fa0525",
            "6b5ac87371fa4d7fa3a7c6e3c4f1fcd0",
            "a16a036b1aa546548b6537860f932335",
            "624e3194f9964e89bf9684cdeea07211",
            "b130ee650d8f4d46954fafe735778d1a",
            "0551978b6170471690f8145537ce3b85",
            "52a6fb9fc1bc47ee879bd81d4a25a0dd",
            "5776f129e5c44339aebfef2d970030aa",
            "61152fc0162b4f51a390435a6864b99c"
          ]
        },
        "id": "jJ-PcXSmrQc6",
        "outputId": "497030ae-0864-4c74-ad37-af44ce695ee3"
      },
      "outputs": [],
      "source": [
        "daily_dfs = {}\n",
        "\n",
        "for basin in tqdm(basins):\n",
        "    netcdf_file = OUTPUT_DIR / \"temp\" / basin / \"combined.nc\"\n",
        "    if not netcdf_file.is_file():\n",
        "        print(f\"Combined netCDF file does not exist at {netcdf_file}\"\n",
        "               \" Make sure to succesfully execute the blocks above\")\n",
        "    df = xarray.open_dataset(netcdf_file).to_dataframe()\n",
        "    # By ERA5-Land conventions, PET, as an upward flux, has a negative sign.\n",
        "    # We change this to a positive sign, which is more common for hydrological applications.\n",
        "    df[\"potential_evaporation\"] = df[\"potential_evaporation\"] * -1\n",
        "\n",
        "    # Disaggregate features that are accumulated over the day into hourly values.\n",
        "    df = caravan_utils.disaggregate_features(df)\n",
        "\n",
        "    # Clip precipitation and SWE to 0. They can have small negative values due to numerical issues.\n",
        "    df.loc[df[\"total_precipitation\"] < 0, \"total_precipitation\"] = 0.0\n",
        "    df.loc[df[\"snow_depth_water_equivalent\"] < 0, \"snow_depth_water_equivalent\"] = 0.0\n",
        "\n",
        "    # Convert ERA5-Land variables into units that are more common for hydrological applications.\n",
        "    df = caravan_utils.era5l_unit_conversion(df)\n",
        "\n",
        "    # Aggregate dataframe to daily resolution in local time\n",
        "    df_daily = caravan_utils.aggregate_df_to_daily(\n",
        "        df=df, \n",
        "        gauge_lat=df_metadata.loc[df_metadata.index == basin, 'gauge_lat'].values[0],\n",
        "        gauge_lon=df_metadata.loc[df_metadata.index == basin, 'gauge_lon'].values[0],\n",
        "        mean_vars=MEAN_VARS,\n",
        "        min_vars=MIN_VARS,\n",
        "        max_vars=MAX_VARS,\n",
        "        sum_vars=SUM_VARS\n",
        "        )\n",
        "\n",
        "    daily_dfs[basin] = df_daily"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "txsYf2Hv4-EG"
      },
      "source": [
        "# Load streamflow data\n",
        "\n",
        "**USER INPUT REQUIRED**\n",
        "\n",
        "Here, you need to add code to load area-normalized daily streamflow data (**unit: mm/d**) that will be merged with the time series from ERA5-Land.\n",
        "\n",
        "You need to do the following:\n",
        "- Load area-normalized streamflow data from any source and make sure it is in daily resolution. The assumption is that the data is in local time so that it matches the ERA5-Land aggregates we computed above in local time. If your streamflow data are not daily averages in local time make sure to adapt the aggregation code above.\n",
        "- Create a pandas DataFrame, called `df_streamflow` that is datetime-indexed and contains one column with streamflow data per gauge/basin.\n",
        "- The name of the column has to match the gauge/basin id, as in the `df_lat_lon` DataFrame and the ERA5-Land data from Earth-Engine.\n",
        "- Missing values should be indicated by `NaN` (e.g. using `np.nan`) and _not_ by negative numbers.\n",
        "\n",
        "Note: It is possible to proceed, even if you do not have streamflow data for a particular gauge. In this case, simply add a column for said basin filled with NaN values. Also note that the length of the streamflow records does not have to match the entire period of the meteorological forcings but can be less."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Load your streamflow data here. See specifications above\n",
        "df_streamflow = ..."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "rbFGEOi-jz-M"
      },
      "source": [
        "### Validate loaded streamflow data\n",
        "Execute this code section to test the validity of your `df_streamflow` DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QCs8mbZ43Z1m",
        "outputId": "e2dfdf8b-1c03-4ca6-8c7e-7802d11b8e77"
      },
      "outputs": [],
      "source": [
        "if not 'df_streamflow' in locals():\n",
        "    raise RuntimeError(\"A variable 'df_streamflow' does not exist\")\n",
        "\n",
        "if not isinstance(df_streamflow, pd.DataFrame):\n",
        "    raise RuntimeError(\"'df_streamflow' is not a pandas DataFrame\")\n",
        "\n",
        "missing_basins = [x for x in daily_dfs.keys() if x not in df_streamflow.columns]\n",
        "if missing_basins:\n",
        "    raise RuntimeError(\"The following basins are missing in the columns of the \"\n",
        "                        f\"DataFrame: {missing_basins}\")\n",
        "  \n",
        "if not isinstance(df_streamflow.index, pd.core.indexes.datetimes.DatetimeIndex):\n",
        "    raise RuntimeError(\"The streamflow needs to have a DatetimeIndex\")\n",
        "  \n",
        "if not pd.infer_freq(df_streamflow.index) == 'D':\n",
        "    raise RuntimeError(\"The frequency of the DatetimeIndex must be daily but \"\n",
        "                        \"isn't or can't be inferred.\")\n",
        "\n",
        "if df_streamflow.lt(0.0).any().any(): \n",
        "    raise RuntimeError(\"DataFrame contains negative values. Make sure to mark invalid \"\n",
        "                       \"streamflow by NaN and not by a negative constant.\")\n",
        "# Make sure the index is called \"date\"\n",
        "df_streamflow.index.name = \"date\"\n",
        "\n",
        "print(\"All tests passed successfully.\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "pTcPvHryr9Gb"
      },
      "source": [
        "# Merge streamflow and ERA5-Land time series\n",
        "\n",
        "Finally, we can merge the streamflow data to the ERA5-Land time series. Note, even if the streamflow records exceed the length of the ERA5-Land period, we only keep the data for which ERA5-Land features exist.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "stX_Y9UsrOdC"
      },
      "outputs": [],
      "source": [
        "for basin, basin_df in tqdm(daily_dfs.items()):\n",
        "    start_date, end_date = basin_df.index[0], basin_df.index[-1]\n",
        "    basin_df[\"streamflow\"] = df_streamflow[basin][start_date:end_date]"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Truncate precision of data to 2 decimal digits\n",
        "\n",
        "The data in Caravan will be shared rounded to two decimal digits. Here, we make sure that we perform the truncation before computing the climate indicies (see box below)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "for basin, df in tqdm(daily_dfs.items()):\n",
        "    daily_dfs[basin] = df.round(2).applymap('{:.2f}'.format).applymap(float)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Compute climate indices\n",
        "\n",
        "Besides the HydroATLAS attributes that are part of Caravan, we also add a couple of climate indices that are derived from the ERA5-Land time series. Most of these are the same attributes provided with the CAMELS US dataset (see [Addor et al.](https://hess.copernicus.org/articles/21/5293/2017/)). The exception are the seasonality, fraction of snow and the moisture index, which are computed following [Knoben et al.](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018WR022913) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "climate_indices = {}\n",
        "for basin, df in tqdm(daily_dfs.items()):\n",
        "    climate_indices[basin] = caravan_utils.calculate_climate_indices(df)\n",
        "df_caravan = pd.DataFrame.from_dict(climate_indices, orient='index')\n",
        "df_caravan.index.name = \"gauge_id\"\n",
        "df_caravan.head()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "4dE_mTOn4nI5"
      },
      "source": [
        "# Add area to metadata and remove from HydroATLAS attributes\n",
        "\n",
        "The basin area is stored in the Earth Engine attribute file. We remove the column from there and drop it from the HydroATLAS attributes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        },
        "id": "kcsgQTTX4zj9",
        "outputId": "0ff57ff5-f3c5-459c-b532-8ad25ec1d06f"
      },
      "outputs": [],
      "source": [
        "# Make sure we only add rows of basins for which we processed time series data\n",
        "df_metadata = df_metadata.loc[basins]\n",
        "\n",
        "# Make sure we only add the lat/lon column\n",
        "drop_cols = [x for x in df_metadata.columns if x not in ['gauge_lat', 'gauge_lon', 'gauge_name', 'country']]\n",
        "df_metadata = df_metadata.drop(drop_cols, axis=1)\n",
        "\n",
        "# Add area to metadata df.\n",
        "df_hydroatlas = pd.read_csv(ATTRIBUTE_FILE, index_col='gauge_id')\n",
        "df_metadata['area'] = df_hydroatlas.loc[basins, 'area']\n",
        "\n",
        "# Remove area from hydroatlas features.\n",
        "df_hydroatlas = df_hydroatlas.drop(['area'], axis=1)\n",
        "\n",
        "# Sort all DataFrames alphanumerically\n",
        "df_metadata = df_metadata.sort_index(axis=0)\n",
        "df_caravan= df_caravan.sort_index(axis=0)\n",
        "df_hydroatlas = df_hydroatlas.sort_index(axis=0)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ZcFTm2tE3ldt"
      },
      "source": [
        "# Save results to disk\n",
        "\n",
        "Having everything processed, we can finally store the results to disk to extend the Caravan dataset. The data is stored in the following structure (to mimic the fold structure of the Caravan dataset) and can be integrated as-is into the original Caravan dataset.\n",
        "\n",
        "- `{OUTPUT_DIR}/`: The outermost folder is the user-defined `OUTPUT_DIR` (see General Configuration section). All resulting data will be stored in this folder, mimicking the folder structure of the Caravan dataset.\n",
        "- `{OUTPUT_DIR}/attributes/{BASIN_PREFIX}/`: This folder will contain three files. The first is `attributes_caravan_{BASIN_PREFIX}.csv`, which contains the climate indices computed above. The second is `attributes_hydroatlas_{BASIN_PREFIX}.csv`, which contain the HydroATLAS attributes as saved by Earth Engine. The third is `attributes_other_{BASIN_PREFIX}.csv`, which contain the metadata information, such as gauge_lat, gauge_lon, gauge_name, country, and area.\n",
        "- `{OUTPUT_DIR}/timeseries`: The per-basin time series data will be stored in this folder. Two options exist: as netCDF file or as csv file. For each option, a different folder is created, `{OUTPUT_DIR}/timeseries/netcdf` and `{OUTPUT_DIR}/timeseries/csv`. In each of these folders, you will find a folder similar to the user-defined `BASIN_PREFIX`. This folder finally contains one netCDF (or one csv file) per basin, called `{gauge_id}.nc` (or `{gauge_id}.csv`).\n",
        "\n",
        "This folder/file structure can be copied directly into the original Caravan dataset (or stored separately, as you prefer)."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "mmQkGtVT6_D0"
      },
      "source": [
        "## Save attributes\n",
        "\n",
        "This block will create two files:\n",
        "\n",
        "- `{OUTPUT_DIR}/attributes/{BASIN_PREFIX}/attributes_hydroatlas_{BASIN_PREFIX}.csv`: The HydroATLAS attributes as saved from Earth Engine.\n",
        "- `{OUTPUT_DIR}/attributes/{BASIN_PREFIX}/attributes_caravan_{BASIN_PREFIX}.csv`: The climate indices computed in this notebook.\n",
        "- `{OUTPUT_DIR}/attributes/{BASIN_PREFIX}/attributes_other_{BASIN_PREFIX}.csv`: Metadata information, such as gauge_lat, gauge_lon, gauge_name, country, and area."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VRlGqLMH6-Hl"
      },
      "outputs": [],
      "source": [
        "attribute_dir = OUTPUT_DIR / \"attributes\" / BASIN_PREFIX\n",
        "if not attribute_dir.is_dir():\n",
        "    attribute_dir.mkdir(parents=True)\n",
        "\n",
        "# Store climate attributes from DataFrame to disk.\n",
        "df_metadata.to_csv(attribute_dir / f\"attributes_other_{BASIN_PREFIX}.csv\")\n",
        "df_caravan.to_csv(attribute_dir / f\"attributes_caravan_{BASIN_PREFIX}.csv\")\n",
        "df_hydroatlas.to_csv(attribute_dir / f\"attributes_hydroatlas_{BASIN_PREFIX}.csv\")\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "9LmcR6aG4ggq"
      },
      "source": [
        "## Save as netCDF\n",
        "\n",
        "Execute the sections below, if you want to store the data in netCDF format. Because netCDF can include metadata information, we will add some metadata before storing to disk. More specifically, we add the following information:\n",
        "\n",
        "- `Units`: Explanation of the physical unit of each variable.\n",
        "- `Timezone`: A string defining the local time zone of this gauge.\n",
        "- `Sources`: A user-defined string (see below) that adds information about the source of the data. Here you can add information about the source (and license) of the streamflow data.\n",
        "\n",
        "If you do not want to store the data in netCDF format, proceed to the section \"Save as csv\"."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "pid2NKkG7sAj"
      },
      "source": [
        "### Metadata information\n",
        "\n",
        "Use the variable `data_sources` to add any information, e.g., about the source of the streamflow data. It is recommended to include the statement of ERA5-Land, especially if you want to share your data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XUyL5-XuHfAE"
      },
      "outputs": [],
      "source": [
        "# Add any information that you like to this string.\n",
        "data_sources = ('All forcing and state variables are derived from ERA5-Land ' \n",
        "                'hourly by ECMWF. Streamflow data ...')\n",
        "\n",
        "if not isinstance(data_sources, str):\n",
        "    raise RuntimeError(\"'data_sources' has to be a string.\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "9AHSa6Le-w_y"
      },
      "source": [
        "### Convert to xarray Dataset, add metadata, and save to disk"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a2UaUye7-jpA"
      },
      "outputs": [],
      "source": [
        "tf = TimezoneFinder()\n",
        "\n",
        "netcdf_dir = OUTPUT_DIR / \"timeseries\" / \"netcdf\" / BASIN_PREFIX\n",
        "\n",
        "for basin, basin_df in tqdm(daily_dfs.items()):\n",
        "    xr = xarray.Dataset.from_dataframe(basin_df).astype(np.float32)\n",
        "\n",
        "    metadata = caravan_utils.get_metadata_info(xr)\n",
        "    unit_info = \"\"\n",
        "    for k, v in metadata.items():\n",
        "        unit_info = unit_info + f\"{k}: {v}\\n\"\n",
        "    xr.attrs[\"Units\"] = unit_info\n",
        "\n",
        "    lat = df_metadata.loc[df_metadata.index == basin, \"gauge_lat\"].values[0]\n",
        "    lon = df_metadata.loc[df_metadata.index == basin, \"gauge_lon\"].values[0]\n",
        "    xr.attrs[\"Timezone\"] = tf.timezone_at(lat=lat, lng=lon)\n",
        "    xr.attrs[\"Sources\"] = data_sources\n",
        "\n",
        "    output_path = netcdf_dir / f\"{basin}.nc\"\n",
        "    if not output_path.parent.is_dir():\n",
        "        output_path.parent.mkdir(parents=True)\n",
        "    xr.to_netcdf(output_path)\n",
        "\n",
        "print(f\"netCDF files stored at {netcdf_dir}\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "dZWmIQGVB_8O"
      },
      "source": [
        "## Save as csv\n",
        "\n",
        "Execute the section below if you want to (additionally) store the data in csv format. Here, we can't add any metadata information, so make sure to share the information about license/data sources in other forms when sharing the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RInQU3CT_95p"
      },
      "outputs": [],
      "source": [
        "csv_dir = OUTPUT_DIR / \"timeseries\" / \"csv\" / BASIN_PREFIX\n",
        "for basin, basin_df in tqdm(daily_dfs.items()):\n",
        "    output_path = csv_dir / f\"{basin}.csv\"\n",
        "    if not output_path.parent.is_dir():\n",
        "        output_path.parent.mkdir(parents=True)\n",
        "    df = basin_df.round(2).applymap('{:.2f}'.format).applymap(float)\n",
        "    df.to_csv(output_path)\n",
        "\n",
        "print(f\"csv files stored at {csv_dir}\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Remove temporary directory"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "shutil.rmtree(str(OUTPUT_DIR / \"temp\"))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "tV9cJ48kKpE_"
      },
      "source": [
        "# End\n",
        "\n",
        "You have reached the end of this notebook. As a result, you should have new data that integrates seamlessly into the Caravan dataset. As the intend of Caravan is to be a dynamically growing, open source dataset, consider sharing your data with the rest of the community. When doing so, make sure that the license of the streamflow data allows further distribution. Currently, the place for sharing data and discussing extensions of Caravan is the [Github discussion forum]() of the Caravan dataset."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Caravan_part2_postprocess_locally.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3.7.13 ('caravan')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.13"
    },
    "vscode": {
      "interpreter": {
        "hash": "16ca22d4d6ad62e956678673fa39b18ef107fac5eae1759b233d19137d7ec387"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0551978b6170471690f8145537ce3b85": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "05f3fefab29a49f3bcd95a32e61c12e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7ceaddba1f97468093be4ebba6a8b1c8",
            "placeholder": "​",
            "style": "IPY_MODEL_9ceb9dcc604f472399c11af374590258",
            "value": "100%"
          }
        },
        "2205d92a1d5446e4926fdbfe1188ed84": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "28bcbfba0dad4138aeafef2fe138c119": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "515a0303d1734242a53a03be63f03a72": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8de72c2c744b46cb84c2f02cecd7098a",
            "placeholder": "​",
            "style": "IPY_MODEL_8044f418241b4b05b95ad611cb215348",
            "value": " 2/2 [00:00&lt;00:00,  1.35it/s]"
          }
        },
        "52a6fb9fc1bc47ee879bd81d4a25a0dd": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5776f129e5c44339aebfef2d970030aa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "61152fc0162b4f51a390435a6864b99c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "61b78d2b8a1e4caab0fc0bb555daebdf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "624e3194f9964e89bf9684cdeea07211": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6b5ac87371fa4d7fa3a7c6e3c4f1fcd0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_52a6fb9fc1bc47ee879bd81d4a25a0dd",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0551978b6170471690f8145537ce3b85",
            "value": 2
          }
        },
        "75089d3f666644fe8ebf82e263683fab": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7ceaddba1f97468093be4ebba6a8b1c8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7f42559288c645d2a0f3960ac05e30bc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8044f418241b4b05b95ad611cb215348": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "831f1ae7ad2a4e0a953ae5e25ee5e711": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d1c5dd72745a4cc5a1e534e40c9148eb",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_61b78d2b8a1e4caab0fc0bb555daebdf",
            "value": 2
          }
        },
        "8a389005ad7b4a55b6b2fc84136ed9b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8de72c2c744b46cb84c2f02cecd7098a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "96927a3d64ae4549a37b6ad07cf1620d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f861974ab8e5439e850005c5c7a9b005",
            "placeholder": "​",
            "style": "IPY_MODEL_8a389005ad7b4a55b6b2fc84136ed9b1",
            "value": "100%"
          }
        },
        "9ceb9dcc604f472399c11af374590258": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9ec4d400994e4034975bd9a9de1a9622": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f416a0e2889d4d0897c4ba8f22fa0525",
              "IPY_MODEL_6b5ac87371fa4d7fa3a7c6e3c4f1fcd0",
              "IPY_MODEL_a16a036b1aa546548b6537860f932335"
            ],
            "layout": "IPY_MODEL_28bcbfba0dad4138aeafef2fe138c119"
          }
        },
        "9fa0de5ba3e54a94a950e0b037969965": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e5735a34bdc846e88a594b3e64b616b1",
            "placeholder": "​",
            "style": "IPY_MODEL_2205d92a1d5446e4926fdbfe1188ed84",
            "value": " 2/2 [00:00&lt;00:00, 24.43it/s]"
          }
        },
        "a16a036b1aa546548b6537860f932335": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_61152fc0162b4f51a390435a6864b99c",
            "placeholder": "​",
            "style": "IPY_MODEL_5776f129e5c44339aebfef2d970030aa",
            "value": " 2/2 [00:00&lt;00:00, 20.03it/s]"
          }
        },
        "b047b0b785084102a774d0d796c74d76": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b130ee650d8f4d46954fafe735778d1a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cb6e1a299d5843df9cbdcf94843ce739": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fd173cc7b1234ab682997ebd3dd964b5",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b047b0b785084102a774d0d796c74d76",
            "value": 2
          }
        },
        "d1c5dd72745a4cc5a1e534e40c9148eb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "da0021cb57ff49ea83c796f0e3bc7297": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_05f3fefab29a49f3bcd95a32e61c12e8",
              "IPY_MODEL_831f1ae7ad2a4e0a953ae5e25ee5e711",
              "IPY_MODEL_515a0303d1734242a53a03be63f03a72"
            ],
            "layout": "IPY_MODEL_75089d3f666644fe8ebf82e263683fab"
          }
        },
        "e230dec250e74ae091daa3da6ca26c07": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_96927a3d64ae4549a37b6ad07cf1620d",
              "IPY_MODEL_cb6e1a299d5843df9cbdcf94843ce739",
              "IPY_MODEL_9fa0de5ba3e54a94a950e0b037969965"
            ],
            "layout": "IPY_MODEL_7f42559288c645d2a0f3960ac05e30bc"
          }
        },
        "e5735a34bdc846e88a594b3e64b616b1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f416a0e2889d4d0897c4ba8f22fa0525": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b130ee650d8f4d46954fafe735778d1a",
            "placeholder": "​",
            "style": "IPY_MODEL_624e3194f9964e89bf9684cdeea07211",
            "value": "100%"
          }
        },
        "f861974ab8e5439e850005c5c7a9b005": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fd173cc7b1234ab682997ebd3dd964b5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
